Exploring comparative marking

UKCLE PDF project

Project team: James Hanlon (Sheffield Hallam University), Michael Jefferson (University of Sheffield), Brian Mitchell (University of Wolverhampton), Mike Molan (London South Bank University)
Project mentor: Jo Coles-Ritson (University of Plymouth)
Project summary: an examination of the incidence of ‘error variation’ arising from the grading of level 1 assessments across four university law schools
Completion date: November 2004
UKCLE funding: £2,600

---

 

This project sought to test the hypothesis that work submitted for assessment by level 1 undergraduate law students is subject to consistent grading across university law schools. It involved a study of the grades awarded to students across four law schools (Sheffield, Sheffield Hallam, South Bank and Wolverhampton).

Download a copy of the final report from the project, An examination of the incidence of error variation in the grading of law assessments at the bottom of the page (RTF file, 63 pages, 1.56 MB).

The research involved a staged comparative marking exercise:

• students at the four law schools undertook a common assessment in a level 1 module, which was internally assessed and graded
• data resulting from this exercise was analysed to provide indicators of error variation
• a group of teachers from a range of institutions undertook a controlled assessment exercise at the University of Wolverhampton on 15 July 2004, involving the marking of a sample of anonymised student scripts in the area of criminal law
• data from this exercise was analysed to provide an insight into the issues of error variation and comparability of standards

Findings

Pilot stage

The views of the researchers at this point, ie at the pilot stage (bearing in mind the relatively limited numbers of markers and the numbers of scripts), may be summarised thus:

• there was no marked difference between markers from old and new universities in the level of marks awarded (the original hypothesis)
• there was no difference between the genders
• there was no difference as to whether the markers were old or young
• there was no difference as to whether the markers were inexperienced or not
• there was some difference between marks of the same examiners after a gap of time
• there was evidence that careful and reasonable markers given the same guidance and the same script could produce differing grades

It was this last point that prompted the research project team to extend the scope of the inquiry further.

Conclusions

1. Intuitive understanding of standards: on the basis of the marking exercises carried out in the early stages of the project, there cannot be said to be an intuitive understanding, amongst those participating in the exercises that made up the research project, of the standards to be applied in respect of the work that was the subject of this study. Not surprisingly, when markers were asked to grade work on the basis of their own intuition without the benefit of assessment criteria and follow-up moderation there was variation in the grades awarded in respect of the work that formed the basis of the exercise.
2. Use of assessment criteria: the use and understanding of criteria did not eradicate marker variation. It is often argued that it is sufficient for members of marking teams to be briefed on the assessment criteria to be applied when marking student work to ensure the uniform application of standards.
3. Moderation of grades: uniform application of standards was only achieved through the use of a rigorous process of moderation, such as was applied in the final stage of the research. This involved all members of the marking team being thoroughly briefed by the author of the questions used to test student knowledge.

Recommendations

1. The major recommendation arising out of this research is the importance of recognition that the process of assessment is adequately resourced. Markers must be given the support and time needed to carry out the process of assessment in a considered manner. This must involve ensuring that they have a full understanding of the assessment criteria to be applied and the outcomes students are required to show competence in.
2. What should also be considered is the guidance to be gained from using statistical tools to assist the process of assessment. The existence of error variation can be detected through the use of what to a statistician is a relatively simple calculation.

Further research

• A question that the research did not answer is the extent to which there is shared understandings of standards across the sector. The project did not, for instance, seek to evaluate the extent to which there was an understanding of the extant guidance on levels of graduate achievement as found in the _Benchmark statement for law_.
• Further research may be necessary into the nature and usefulness of marking criteria and guidance for internal and external markers, and the extent to which they assist in the prevention of assessment grade error variation.
• The lack of a highly defined prescriptive approach to the coverage of the Foundations of legal knowledge in law degrees, and the extent to which this hampers comparisons of marking standards, also provides a useful avenue for further research.

Staff development

We are similarly confident of the benefits to be gained from markers from different institutions meeting to undertake grading exercises. It is through such exercises, which attracted participants at all stages of the research project, that cross-institution appreciation of standards can be generated, and we recommend that they become a standard feature of the work of UKCLE as it seeks to disseminate good practice in the teaching and assessment of law.

Outcomes

• An examination of the incidence of ‘error variation’ in the grading of law assessments – final report (RTF file, 63 pages, 1.56 MB). Project findings were presented at LILI 2005.

---

Last Modified: 6 July 2010